Occlusion Handling and Image-based Lighting using Sliced Images in 3D Photo Collections

This paper presents novel methods to handle potential occlusion problems and to render realistically the lighting of embedded virtual 3D objects in photo-based 3D worlds. These 3D worlds are called 3D Photo Collections. The first focus of this work handles potential occlusion problems between image parts of photos and the embedded virtual objects by dividing photos into 3D segments, which we call slices. The segmentation of the photos is done by a gradient-based contour tracing algorithm, which divides a photo into several contours. These contours are used in the 3D world as slices. Each slice will be moved to a depth position estimated by the provided spatial interdependency between photos of the 3D Photo Collection. This results in several slices at different depth positions per photo. We call the divided photo SlicedImage. The second focus of this paper is concentrated on the realistic rendering of the illumination of embedded virtual objects by extracting the environmental lighting of the real scene from the generated Sliced Images. For fully automated generating of image-based 3D worlds, 3D Photo Collections (generated by software like Microsoft Photosynth or Google Street View) are used. These 3D Photo Collections provide spatial interdependency between photos, which will be used as input data for our algorithms to compute the Sliced Images.